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Overview: Program Evaluation in the College 
of Education at Michigan State University 

Program evaluation in the College of Education at Michigan State 
University is an ongoing, cooperative endeavor involving numerous groups and 
individuals. This report begins with a brief description of the roles of the 
contributing members. It then describes ongoing activities across the three 
major phases of program evaluation - (a) data collection, (b) data analysis and 
reporting, and (c) internal program reviews - 

PARTICIPANTS 

Graduate and Undergraduate Educat 4^n Policy Committees : The college 
bylaws call for the Graduate Education Policy Committee (GEPC) to review arid 
evaluate all graduate programs offered by the College of Education and 
stipulate that the Undergraduate Education Policy Committee (UEPC) should 
assume the same responsibilities in regard to undergraduate programs. Both 
committees include two elected representatives from each department, a 
department chair, and a representative of the Dean's Office; 

The criteria and procedures that serve as the framework for periodic 
reviews of graduate programs are described in a GEPC policy statement adopted 
in January,. 1984. The first review of this type was conducted in 1984-85 and 
focused on graduate programs in K-12 administraticn. A second review is 
currently underway for three programs offered by the Department of CbUhseiing, 
Educational Psychology and Special Education. 

During the 1984'85 academic year, the Undergraduate Education Policy 
Cbminittee developed a statement of guidelines and procedures for periodic 
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reviews of undergraduate programs. These Were approved as a formal UEPC policy 
statement in May, 1985. It is anticipated that these reviews will begin in the 
198(5-87 academic year. 

Office of Program EvaluaMon : In 1980, the Dean of the College of 
Education ctfeated the Office of Program Evaluation (OPE) to coordinate the 
evaluation of all undergraduate and graduate pfograis offered by the College, 
its purposes include: 

(1) to provide a substantive information base for faculty 
deliberations focusing on the continued development and 
improvement of programs, including... 

(a) periodic program reviews monitored by the Graduate and 
Undergraduate Education Policy Committees, and 

(b) ongoing program reviews conducted by program faculty 
and coordinators . 

(2) to provide technical assistance or other forms of support for 
individuals engaged in program evaluation or related research 
activities 

(3) to contribute to the research on teacher education. 
Evaluators of Individual Unde rgraduate Pro grams ! In the fail of 1981, 

[ichigan State introduced four new teacher preparation prog^ami as alternatives 
:o the traditional program. At that same time, five faculty members accepted 
.ppointments (1/4 time each) as evaluatbirs of each of the five teacher 
reparation programs. These individuals (1) design and conduct evaluation 
tudies that are unique to their programs (e.g., studies of the match between 
rogram goals and instruction), and (2) work with other evaluators in designing 
tid using program evaluation instruments and procedures that are common to all 
rbgrams (e.g., entry-level questionnaires and interviews). 

UtiderKraduate Program Evalua t ion Committee : Soon after the new teacher 
reparation programs were introduced, the Undergraduate Program Evaluation 
ommittee (UPEC) was created. Its purpose is to provide continuity among those 
sspbnsible for the evaluation of undergraduate programs. The UPEC includes 
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members of the Office of Program Evaluation staff, the Assistant Dean for 
Teacher Education, and the five individuals who serve as evaluatbfs of 
individual programs. The committee plays a central role in the design 8f 
program evaluation instruments and procedutreis arid provides a forum for 
discussing evaluation issues and findings. 

O^her Researchers and R vatuators : In addition to the ohgbirig evaluation 
assignments noted above, other MSU faculty conduct independent research or 
evaluation studies that contribute to the inforinatibn base considered in 
program reviews. Examples of studies of this type that have been completed 
since 1980 include: (a) across -program evaluations of three Undergraduate 
courses - educational psychology, reading methods, and social studies methods; 

(b) follow-up studies of graduates of two M.A. programs in special education; 

(c) an ethnographic analysis of students' acquisition of professional 
knowledge; (d) an investigation of the influence of MSU's teacher preparation 
programs on teacher candidates' deveidpraent of ah international perspective; 
and, (e) a study of how prospective teachers leairn to make interactive 
decisions . 

College Administrators : Various college administrators also play an 
active role in program evaluation. The Associate Dean for Admin'.strative 
Services oversees the program evaluation function in the College. The 
Assistant Deati for Teacher Education is actively involved in the evaluation of 
undergradua^te programs and serves as the Dean's representative on the 
Undergraduate Program Evaluation Committee. Department chairs allocate faculty 
rescarces to program evaluation arid oversee the implementation of 
recommendations resulting from program reviews. 

DATA COLLECTION 

For a variety of reasoiiS, infdrraatibri collected by the Office of Program 
Evaluation and other evaiuators iri support of reviews of undergraduate programs 
is more comprehensive than that collected iri support of reviews of graduate 
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programs. The undergraduate data eollectibh piah trades the progress of 
teacher candidates fr5m the t±5e they enter a program Until five 5r six years 
after they graduate. The graduate daEa collection plan, on the othef hand, 
provides a crbss-sectibhal rather than a longitudinal data base and focusis 
primarily ori- fbllbw^up studies of program graduateis. Therefore, the tv/o data 
collection mbdels will be cbnsidered independently. 

eonceptual Framework for Undergraduate Programs : The conceptual framework 
for the evaluation bf undergraduate teacher preparation programs is illustrated 
in Figure 1 on page 5. The basic question that drives data collectibn at the 
undergraduate level is, "How do students change as they progress thrbUgh each 
of Michigan State's five teacher preparation programs?" In more specific 
terms, "What changes occur between program entry and program exit across three 
general categories of student oUtcbmes ^ (a) acquisition of professional 
knowledge; (b) competence in teaching performance; and, (c) educational 
orientations and beliefs?" Subquestions include the following: "Do changes in 
student outcomes vary in predictable ways as a function of the alternative 
teacher preparation program in which the candidate participated?," "Do these 
changes endure over time?« Although the primary fbcus is on student outcomes 
(i.e., prbducts), process variables are also considered. The basic question 
that guides the analysis of process variables is, "To what extent have stated 
program goals and bbjectives been adequately addressed in the prbgram?" 

The c&ntral questibns in the undergraduate program evaluation plan Were 
identified by the Undergraduate Program Evaluation Committee during its initial 
yrear of operation. At that time, tentative agreements were also reached 
regarding who would assume major responsibility for addressing each question - 
the Office of Program Evaluation, individual prbgram evaluators , or both. 
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Figure I . CbticepCuai Framework iEor Undergraduate Prbgram Evaluation 
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These ihitiii aireenients were suiSiflarized in a dbdUmeSt entitled, "Undergraduate 
Program Evaluation eommittee: Futlctidnai Orgariizatibii« (UPEC, 1982). Since 
that time, the specific ways in which the central questions have been addressed 
have been shaped by three major sources of ittpUt: (a) interviews involving 
representatives of various target audiences, (b) ongoing deliberations of the 
Undergraduate Program Evaluation Committee and (c) ongoing intiractioni with 
program faculty arid cbdrdihators . 

in target audience interviews conducted in 1982. participants were asked 
to describe information they and their colleagues would be likely to Use. 
Target audiences that were represented included: p^ograi faculty and 
administrators, college administrators, chairs of relevant College committees , 
alumni. K-12 school administrators, and members of the Michiian Department of 
Education. The results played a major role in the UPEC's identification of 
data that is collected across ail programs. [See "Undergraduati Program 
Evaluation Model: Data Collection Activities Cbmrabh to All Programs" (UPEC, 
1983)] . 

Data^eaee^^n Instruments : The undergraduate data collection plan is 
based on two distinct types of data collection instruments. One subset 
includes instruments selected o^ developed by program faculty to provide 
measures of particular concern to individual programs (e.g., the Myers-Briggs 
Type Indicator is used in two of the five programs). The other subset includes 
instruments- developed by the Undergraduate Program Evaluation Committee to 
provide a common, comparative baseline for interpreting the results for each 
program. Each instruiiient in this subset was developed with extensive input 
from program faculty. 

At present, the set of common instruments includes three questionnaire 
surveys and two interview schedules - ah entry questionnaire survey and 
interview schedule, an exit questionnaire survey and interview schedule, and a 
short- aerni follow-up questionnaire survey. A common schedule has also been 
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developed for interviewing stUdehts wh8 have eieeted to discbhtiriUe their 
participstibn in a teacher preparatibh p^8grain prior to graduation. tJithih the 
next two years, three ihstrumehts will be added to the list of surveys that are 
common to all programs - a short term follow-up interview schedule, a 
questionnaire surVey of supervisors of program graduates, and a long-term 
follow-up questibrihaire . 

Figure 2 on the next page provides an overview of topics that are cbvered 
in each instrument and highlights the repeated measures that are a part bf the 
longitudinal design. As this figure indicates, the five instruments that are 
common to all programs provide a diverse array of information. eoilectively , 
they provide a fairly clear sense of whb bur students are and why they have 
chbseh teaching as a caieei . Some sections bf each questionnaire and each 
interview also provide data related to the three majbr categories of student 
outcomes described earlier in this Report. This set bf instruments is the 
primary data source fbr analyses of chahgei in orientations to teaching and 
educational beliefs. It also provides self- ratings of teaching perfbrraahce and 
perceptions of sources bf prbfessibnal knowledge heeded for teaching. 
Therefore, analyses focusing bri the acquisition of professional knowledge and 
changes in teaching performance draw upbh these sources of information as wall 
as the data provided by observation instruments and written examinations 
develbped by the faculty in each program. 

Sampling Plan : A sampling plan was devised ih 1984 to: (a) make the data 
cbllectioh phase of program evaluation more manageable, and (bj provide an 
brgahizational structure for completing data analyses. As illustrated ih 
Figure 3 bn page 9, a sample, rather than a census of all teacher candidates i 
completes the full set 5f longitudinal instruments - entry questionnaires and 
interviews, ixit questionnaires and interviews, and follbw^up questionnaires 
and interviews . 
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the sample is determined by a three year cycle plan. All students who 
enroll in TE 206. during the 1985-86 academic year will serve as the designated 
sample for cycle 1, all who enroll in TE 200 during the 1988-89 academic year 
will constitute the sample for cycle 2, arid so forth. All students in the 
designated sample will be asked to complete four questionnaires - entry, exit, 
short-term and long-term follow-up surveys . A randomly selected subsample will 
also be asked to participate in three interviews - entry, exit, and follow-up. 
Students who inroll in TE 200 during an off-year (e.g., 1986-87 or 1987-88) 
will complete only the exit and short-term follow-up questionnaires. 

Evalua^on of Graduate- Programs : As noted earlier, comprehensive program 
reviews monitored by the Graduate Education Policy Committee play a central 
role in the evaluation of graduate programs. Data collection activities 
sponsored by the Office of Program Evaluation or other evaluators working at 
this level focUs primarily on the provision of information that will contribute 
to these reviews. 

Data considered in the review of M.A. programs include follow-up studies 
of graduates of M.A. programis . These are coordinated by the Office of Program 
Evaluation and are conducted during the year in which a program is reviewed by 
the GEPC. M.A. follow- ap questionnaires are tailored to the unique 
characteristics of each program and are mailed to a census of all who graduated 
from the program during the preceding five years . 

Data considered in the review of Ed. /Ph.D. programs are derived from 
three sources: (a) a survey administered by the MSU Graduate School at the 
time candidates complete their programs, (b) information included in University 
records or compiled by the Student Affairs Office of the College, arid (c) 
follow-up studies of program graduates conducted once every three years by the 
Office of Program Evaluation. 

The Graduate Education Policy Committee, its program review committees, 
and other members of the faculty have a major voice in the design of M.A. and 
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Ph.D. follbw-up surveys: Topics considered in brie or both of these surveys 
include: employment histories, indices of job satisfaction, descriptions of 
professional accomplishments, "program ratings, critiques of specific features 
of a program (e.g., comprehensive examinations), and recommendatibris for 
program iraprbverhehts . 

DATA ANALYSES AND REPORTING 
In addition tb structuring data collection activities, the three year 
cycle plan provides an brgariizatibrial structure for data analyses and reporting 
at both the graduate and Undergraduate levels. As shown in Figure 3, data 
analyses during the first year bf the three year cycle focus on (1) 
characteristics of students entering teacher preparation programs and (2) 
follow-up studies of graduates of Ed. S. /Ph.D. programs. Analyses during the 
second year focus on (1) the results of shbrt- and long-term follow-up studies 
of alurani of undergraduate programs and (2) investigations of the extent to 
which teacher candidates acquire the profess ibrial knbwledge and teaching skills 
described in undergraduate program goals. Attentibri in the third year shifts 
to characteristics of students at the time they cbraplete an undergraduate 
program and include longitudinal analyses of changes in edUcatibnal 
brientations and beliefs. During each of the three years in the cycle the OPE 
alsb analyzes data from follow-up studies of graduates of the K.A. prbgiraras 
being reviewed that year by the iSraduate Education Policy Committee. 

The data frbra questionnaire surveys are analyzed in two distinct phases, 
in the initial phase, simple frequency distributions of responses to each 
questionnaire item are generated. These distributions are reviewed by the 
committees who are most likely to use this infbrraatibri (e.g.. Undergraduate 
Program Evaluation Committee; Undergraduate or Graduate Education Policy 
Committees; program review committees established by the GEPC br UEPC) . The 
primary intent of these preliminary reviews is to identify secondary analyses 
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that will provide useful iriforraatidh fbr program faculty. During the second 
phase of data analysis, secondary analyses identified in phase one are 
completed by the Office of Prbgrain Evaluation staff and a summary report based 
oh both phases of data analyisis is prepared. 

The analysis of undergraduate istudent interviews also occurs in two 
phasis. In phase one, evaluators of individual programs prepare a sununary of 
results for their program. These results are then discussed by the UPEC in an 
attempt to determine what across -program comparisons , if any, are likely to 
provide useful information for program faculty; These cdmparisons are then 
completed and reported by the Office of Program Evaluation staff. 

Reporting Results : Reports of program evaluation findings are addressed 
to one of three distinct audiences - survey participants, program faculty and 
administrators, and/or external audiences. The first report for a given data 
set highlights some of the preliminary findings and is circulated to suir^ey 
participants. The second report is prepared fbr program faculty. it provides 
a more' comprehensive summary and is based on the results of secondary analyses 
of the data as well as simple frequency distributions. 

When appropriate, short reports are also prepared to highlight evaluation 
findings of general interest to faculty, administrators or ilumni. These are 
circulated to alumni via the College of Education's Alum ni N e wslet -te^ or to the 
faculty through the College of Education Update . 

Program evaluators also contribute to the research literature in teacher 
education. OPE staff and individual program evaluators make frequent 
presentations at national and regional conferences and occasionally prepare 
manuscripts for refereed journals. The Office has also created its own 
publication series. 
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During the 1984-85 academic year, for exainple, members of the OPE staff 

and individual program evaluatbrs presented papers at the Mid-Western 

Educational Research Association ieeting in ehicagb; the joint meeting of the 

Evaluation Research Society and the Evaluation Network in San Francisco; the 

annual meeting n£ the Association of Teacher Educators in Las Vegas; the annual 

meeting of the American Association of Goiieges for Teacher Education in 

Denver; and, the annual, meeting of the American Educational Research 

Association in Chicago. During that same year, one manuscript was published in 

a refereed journal (Book, Freeman, & firousseau, 1985) and four papers were 

added to the OPE's publication series (e.g., tittle, 1984). 

REVIEW OF PROGRAM EVALUATION FINDINGS 

Faculty responsible for the design and implementation of program 

evaluation at MSU recognize that the program ivaluaticn effort will succeed 

only if the info^iation that is collected and reported makes a iubstantive 

contribution to deliberations focusing on the cbritiriued development and 

improvement of programs. Formal and informal communication networks have 

therefore been established to provide systematic communicatibn of program 

evaluation findings to program faculty and administrators. 

Qngding Reviews: The likelihood that program evaluation findings will 

play a prominent role in program development and iip^ovement is enhanced by the 

fsct that the formal commUriicatioh network provides for (1) immediate reviews 

of specific sets of findings and (2) periodic, comprehensive reviews of the 

complete evaluation data base for each program. 

Various college committees provide the forums for immediate reviews of 

specific sets of evaluation findings. Reports prepared for undergraduate 

program faculty, for example, are initially reviewed in a joint meeting of the 

Undergraduate Program Evaluation Committee arid the individuals who serve as 

coordinators of the five teacher preparatibri programs. When appropriate, the 

coordinators and/or individual program evaluatbrs then present pertinent 

findings and/or full reports to the program's faculty. 



^ 17 

ERIC ' 



It 



Periodic, r€omp^rehens"j:ve Reviews : As rioted earlier, cblie^e bylaws 
stipulate that the Graduate and Undergraduate Education Policy eommittees are 
reisponsible for reviewing and evaluating all programs offered by the College. 
Progriam review conimittees established and monitored by the GEPC or UEPC provide 
the stage fc5t periodic reviews of the complete evaluation data base for each 
prograffi. The UEPC plan stipulates that every undergraduate program will be 
reviewed at least once ivery six years; the GEPC plan calls for more" frequent 
reviews of graduate programs, perhaps as often as once every three years. 

In accord With GEPC or UEPC policies, program review committees 
contemplate program modifications and improvements in such areas as: (a) 
admissions standards and procedures, (b) academic advising and other forms of 
support for students, (c) the program's curriculum, (d) faculty and 
administrative assignments, (e) program evaluation procedures, and (f) program 
resources . 

The Office of Program Evaluation and individual prograi evaluators provide 
the program review committee with copies of all pertinent evaluation reports. 
When requested to do so, the OPE also supplements these reports with secondary 
analyses of the data. This and other information compiled by members of t^e 
program review committee provides a substantive data base for the committee's 
deliberations. GEPC procedures also call for an external reviewer to analyze 
current program practices and to recommend program modifications. Provision of 
an external reviewer is strongly encouraged, but nbt required, in the UEPC 
plan. 

The products of the review process include reports prepared by the program 
review committee, the external evaluatbr, and the GEPC or UEPC. Reports 
prepared by prograS Review committees include: (a) recommendatiShs for program 
modifications and improvements and (b) a timeline for reporting subsequent 
progress in implementing these reconunendatibhs . 
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To date, the (3EPC hai aSmpieted a review of graduate prbg^aii in k-12 
admitiistratibh; it is currently Overseeing reviews of three graduate p^o^irams 
offered by the Department of Counseling, Educational Psychology, and Special 
Education. It is anticipated that UEPG Reviews of undergraduate programs will 
begin in the 1936-87 academic year. 
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